I. Field of the Invention
This disclosure relates generally to systems, apparatus and methods for minimizing power consumption of a depth sensor on a mobile device, and more particularly to enabling a depth sensor only when sensor measurements indicate images are inadequate for pose calculation.
II. Background
Motion estimation and trajectory estimation of a mobile device using available sensors have a wide range of important applications. Motion estimation over a short period (e.g., a few milliseconds) is important in de-blurring and super-resolution. Trajectory estimation over an extended period (e.g., a few seconds) is useful in augmented reality (AR), short-range navigation and location estimation. Low-light situations affect both motion estimation and trajectory estimation.
Applications often use inertial sensors for these situations because an inertial sensor measures motion at a higher rate than does a camera. Inertial sensors, however, have inherent biases and other sources of error that add challenges when performing “dead reckoning” without regular bias correction. That is, updating of the bias states of an inertial measurement unit (IMU) need continuous pose estimates. A mobile device may compute a six-degrees-of-freedom (6DoF) pose trajectory using inertial sensor measurements. A 6DoF pose computation considers up-down (heave), right-left (sway) and forward-backward (surge) translation movement as well as forward-backward tilt (pitch), left-right swivel (yaw) and side-to-side pivot (roll) rotation movement. The mobile device may integrate this movement for several 100 milliseconds.
In an example, assume a mobile device includes no pose correction to the IMU, thus providing biased inertial sensor measurements. Integrating such inertial sensor measurements over the short term and long term leads to errors that adversely affect results. For example, assume a 10-frame per second (fps) frame rate of video leads to an exposure time of 100 milliseconds (ms) for each image in the video. Assume further that a 6DoF pose of the mobile device is computed using only inertial sensor measurements and without correcting inertial sensor measurements from the IMU. Such computations will give rise to both pose errors and outdated bias estimates for the IMU.
If inertial sensor error states are periodically corrected with measurements from a non-IMU device, such as a camera, a mobile device may make both better motion estimates and trajectory estimates. Furthermore, the inertial sensor error states may be corrected with such devices. Such non-IMU devices, however, are not continuously available. For example, during periods of camera outage or long image exposure in a visual-inertial tracking system, a camera may not provide pose corrections and inertial-state corrections frequently enough. Another non-camera device may fill this gap to better maintain pose and inertial-state corrections. Such pose and inertial-state corrections may also enhance operation of a recursive filter, such as an extended Kalman filter (EKF), commonly employed to maintain estimates of pose and inertial states.
Some visual-inertial tracking systems that include tracking applications use both inertial sensor measurements from an inertial sensor and images from a camera. Infrequent camera updates, however, may lead to corrupt inertial error states. In turn, the corrupt inertial error states may lead to periods of camera tracking outages due to inaccurate motion prediction. Without assistance from another device during these outages, motion integrated from inertial sensor measurements may be susceptible to bias drift. Furthermore, filter states as a whole may diverge in an unbounded fashion from their true values.
For example, if a camera pose is not available to update inertial states for 6 frames, in a 30-fps video stream, 200 milliseconds (ms) may pass where no correction of states from an non-camera device may occur. During this 200-ms period, an EKF pose estimates and inertial state estimates may drift away from their true values leading to a failure of the EKF. With such a failure, a user application that depends on the EKF inertial state estimates may similarly fail.
In order to alleviate these shortcomings prevalent in visual-inertial tracking systems, embodiments provide pose corrections to an EKF during periods of camera outages when image-based pose measurements are unavailable for longer than 100 ms (e.g., >150 ms). Such pose estimates from non-camera devices may significantly improve the robustness of visual-inertial tracking systems by providing corrective measurements to the EKF. Embodiments described herein provide a correction mechanism from a depth sensor as a non-camera device.